Method and apparatus for controls based on concurrent gestures

ABSTRACT

In an embodiment, a number of parallel gestures are detected, in a particular area on a touch screen interface of a device. A command is identified based at least on the parallel gestures and an action associated with the command is performed.

CROSS REFERENCE TO RELATED APPLICATIONS; BENEFIT CLAIMS

This application is a Divisional of U.S. patent application Ser. No.12/986,060, filed Jan. 6, 2011, the entire contents of which is herebyincorporated by reference for all purposes as if fully set forth herein.The applicant(s) hereby rescind any disclaimer of claim scope in theparent application(s) or the prosecution history thereof and advise theUSPTO that the claims in this application may be broader than any claimin the parent application(s).

RELATED APPLICATIONS

This application is related to application Ser. No. 12/986,054, filed onJan. 6, 2011 and titled “Method and Apparatus for Gesture-BasedControls.”

FIELD OF THE INVENTION

The present invention relates to the use of gestures. Specifically, theinvention relates to gesture-based controls for multimedia content.

BACKGROUND

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

Multimedia content such as web pages, images, video, slides, text,graphics, sound files, audio/video files etc. may be displayed or playedon devices. Commands related to playing or displaying of content ondevices may be submitted by a user on the device itself or on a separatedevice functioning as a remote control.

For example, a user may select a button on a remote control to play,pause, stop, rewind, or fast-forward a video being displayed on atelevision.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1 is a block diagram illustrating an example system in accordancewith one or more embodiments;

FIG. 2 illustrates a flow diagram for detecting a gesture in accordancewith one or more embodiments;

FIG. 3 illustrates an example interface in accordance with one or moreembodiments;

FIG. 4 shows a block diagram that illustrates a system upon which anembodiment of the invention may be implemented.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

Several features are described hereafter that can each be usedindependently of one another or with any combination of the otherfeatures. However, any individual feature might not address any of theproblems discussed above or might only address one of the problemsdiscussed above. Some of the problems discussed above might not be fullyaddressed by any of the features described herein. Although headings areprovided, information related to a particular heading, but not found inthe section having that heading, may also be found elsewhere in thespecification.

Example features are described according to the following outline:

1.0 OVERVIEW 2.0 SYSTEM ARCHITECTURE 3.0 GESTURES 4.0 GESTURE AREA(S)5.0 COMMANDS 6.0 DETECTING A GESTURE WITHIN A GESTURE AREA 7.0 EXAMPLEGESTURES AND COMMANDS 8.0 REMOTE CONTROL USE EXAMPLES 9.0 EXAMPLEEMBODIMENTS 10.0 HARDWARE OVERVIEW 11.0 EXTENSIONS AND ALTERNATIVES

1.0 OVERVIEW

In an embodiment, a gesture is detected in a particular area of a touchscreen interface on a device. The gesture may not necessarily select ormove any visual objects within the particular area. For example, thegesture may be detected in a blank box, on top of a video, on top ofinstructional information for performing gestures, etc. A video playbackcommand associated with the gesture may be identified, and an actioncorresponding to the video playback command may be determined. Theaction may then be performed on the same device that detects thegesture. The action may be performed on a different device that iscommunicatively coupled with the device that detects the gesture.

In an embodiment, multiple input instruments (e.g., multiple fingers)may be used concurrently to perform parallel or identical gestures on atouch screen interface. Based on the number of gestures that aredetected, an action may be selected. For example, the number of gesturesmay also be used to select a particular item from a menu or may be usedto identify a command.

Although specific components are recited herein as performing the methodsteps, in other embodiments agents or mechanisms acting on behalf of thespecified components may perform the method steps. Further, althoughsome aspects of the invention are discussed with respect to componentson a system, the invention may be implemented with componentsdistributed over multiple systems. Embodiments of the invention alsoinclude any system that includes the means for performing the methodsteps described herein. Embodiments of the invention also include acomputer readable medium with instructions, which when executed, causethe method steps described herein to be performed.

2.0 SYSTEM ARCHITECTURE

Although a specific computer architecture is described herein, otherembodiments of the invention are applicable to any architecture that canbe used to perform the functions described herein.

FIG. 1 is a block diagram illustrating an example system (100) inaccordance with one or more embodiments. The example system (100)includes one or more components that function as content sources, touchscreen interface devices, multimedia devices (e.g., devices that playaudio and/or video content), and/or content management devices. Specificcomponents are presented to clarify the functionalities described hereinand may not be necessary to implement one or more embodiments. Each ofthese components are presented to clarify the functionalities describedherein and may not be necessary to implement one or more embodiments.

Components not shown in FIG. 1 may also be used to perform thefunctionalities described herein. Functionalities described as performedby one component may instead be performed by another component.

An example system (100) may include one or more of: an input device(110), a multimedia device (140), and a data repository (150). One ormore devices shown herein may be combined into a single device orfurther divided into multiple devices. For example, the input device(110) and the multimedia device (140) may be implemented in a singledevice. The multimedia device (140) may be configured to play audioand/or video content. The multimedia device (140) may be configured todisplay one or more still images. In another example, an input device(110) may be used as a remote control detecting gesture-based commandsrelated to content being displayed on a separate multimedia device(140). The input device (110) may communicate directly with themultimedia device (140) or may communicate with an intermediate device(not shown). The intermediate device may, for example, function as acontent source for the multimedia device (140) or a media managementdevice. A network bus (102) connecting all components within the system(100) is shown for clarity. The network bus (102) may represent anylocal network, intranet, Internet, etc. The network bus (102) mayinclude wired and/or wireless segments. All components (shown ascommunicatively coupled) may not necessarily be communicatively coupledto all other components within the system (100).

In an embodiment, input device (110) may include a touch screeninterface (115) configured to detect one or more gestures, as describedherein. Input device (110) may be configured to detect a gesture, a pathof a gesture, a speed of a gesture, an acceleration of the gesture, adirection of a gesture, etc.

In one example, input device (110) may include a resistive system wherean electrical current runs through two layers which make contact atspots/areas on the touch screen interface (115) that are touched. Thecoordinates of the contact points or contact spots may be compared togesture information stored in a data repository (150) to identify agesture performed by a user on the touch screen interface (115). Inanother example, input device (110) may include a capacitive system witha layer that stores electrical charge, a part of which is transferred toa user where the user touches the touch screen interface (115). Inanother example, input device (110) may include a surface acoustic wavesystem with two transducers with an electrical signal being sent fromone transducer to another transducer. Any interruption of the electricalsignal (e.g., due to a user touch) may be used to detect a contact pointon the touch screen interface (115). For example, input device (110) maybe configured to first detect that an initial user touch on a visualrepresentation, of the data, displayed on the touch screen interface.

In an embodiment, input device (110) may include hardware configured forreceiving data, transmitting data, or otherwise communicating with otherdevices in the system (100). For example, input device (110) may beconfigured to detect a gesture performed by a user and perform a videoplayback action associated with the gesture. In another example, inputdevice (110) may include functionality to transmit information (may bereferred to herein as and used interchangeably with “metadata”)associated with the gesture. For example, input device (110) may beconfigured to transmit information comprising a chronological sequenceof detected contact points on the touch screen interface (115).

In an embodiment, input device (110) may include one or more of: ReadOnly Memory (ROM) (206), a Central Processing Unit (CPU), Random AccessMemory (RAM), Infrared Control Unit (ICU), a key pad scan, a key pad,Non-Volatile Memory (NVM), one or more microphones, a general purposeinput/output (GPIO) interface, a speaker/tweeter, a keytransmitter/indicator, a microphone, a radio, an Infrared (IR) blaster,a network card, a display screen, a Radio Frequency (RF) Antenna, aQWERTY keyboard, a network card, network adapter, network interfacecontroller (NIC), network interface card, Local Area Network adapter,Ethernet network card, and/or any other component that can receiveinformation over a network. In an embodiment, input device (110) may beconfigured to communicate with one or more devices through wired and/orwireless segments. For example, the input device (110) may communicatewirelessly over one or more of: radio waves (e.g., Wi-Fi signal,Bluetooth signal), infrared waves, over any other suitable frequency inthe electro-magnetic spectrum, over a network connection (e.g.,intranet, internet, world wide web, etc.), or through any other suitablemethod.

In an embodiment, input device (110) generally represents any devicewhich may be configured for detecting a gesture as user input. A user(includes any operator of input device (110)) may perform a gesture bytouching the touch screen interface (115) on the input device (110). Forexample, a user may perform a gesture by tapping the touch screeninterface (115) with a finger or sliding a finger on the touch screeninterface (115).

For clarity, examples described herein may refer to a particular inputinstrument (e.g., a user's finger) to perform gestures. However, anyinput instrument including, but not limited to, a stylus, a user'sfinger, a pen, a thimble, etc. may be used to perform gestures inaccordance with one or more embodiments.

Gestures relating to touching or making contact with the touch screeninterface (115), as referred to herein, may include hovering over atouch screen interface (115) with a finger (or other input instrument)without necessarily touching the touch screen interface (115) such thatthe touch screen interface (115) detects the finger (e.g., due totransfer of electrical charge at a location on the touch screeninterface (115)).

3.0 GESTURES

In an embodiment, a tap gesture may be performed by touching aparticular location on the touch screen interface (115) and thenreleasing contact with the touch screen interface (115). A tap gesturemay be detected by detecting a contact to a touch screen interface (115)at a particular location followed by detecting that the contact isreleased.

A tap gesture may refer to a gesture performed using one or morefingers. For example, a two-fingered tap may be performed by using twofingers to concurrently touch two locations on a touch screen interface(115) and thereafter release contact with the touch screen interface(115). A two-fingered tap may be detected by concurrently detectingcontact at two locations on the touch screen interface (115) followed bya release of the contact.

In an embodiment, a slide gesture may include any motion in which a userslides one or more fingers on the surface of the touch screen interface(115). Examples of a slide gesture include flick gestures, swipegestures, or gestures involving moving a finger along any path on thetouch screen interface (115). The path may be closed shape such as acircle or square where the start and end points are the same or an openshape such as a right angle where the start and end points aredifferent. Examples of paths include, but are not limited to, a straightline, a curved line, a circle, a square, a triangle, an angle, etc.

In an embodiment, a flick gesture may be performed by touching aparticular location on the touch screen interface (115) of the inputdevice (110) with a finger (or any other item, e.g., a stylus), andsliding the finger away from the particular location while maintainingcontact with the touch screen interface (115) for a portion of thesliding action performed by the user and continuing the sliding actioneven after contact with the touch screen interface (115) has ended. Inan embodiment, the touch screen interface (115) may be configured todetect the proximity of the finger after physical contact with the touchscreen interface (115) has ended.

For example, the user may release contact with the touch screeninterface (115) while still moving the finger in the direction of thesliding action even though additional surface area of the touch screeninterface (115), in the direction of the sliding action, may beavailable to continue the sliding action while maintaining contact.

In another example, a flick gesture may involve a user touching aparticular location on the touch screen interface (115) of input device(110) and then sliding the finger, while maintaining contact with thetouch screen interface (115), beyond the edge of the touch screeninterface (115). Accordingly, the user may maintain contact with thetouch screen interface (115) (e.g., with a finger) until the fingerreaches the edge of the touch screen interface (115) and continue amotion in the same direction past the edge of the touch screen interface(115).

A user performing a flick gesture may continue the sliding action afterreleasing contact with the touch screen interface (115). Input device(110) may detect that contact between a finger and the touch screeninterface (115) was released as the finger was still moving based on aduration of contact with the touch screen interface at the last contactpoint. The detected release while the finger is moving may be determinedto be a flick gesture.

In an embodiment, a swipe gesture may be performed by touching aparticular location on the touch screen interface (115) of the inputdevice (110) with a finger and sliding the finger away from theparticular location while maintaining contact with the touch screeninterface (115) during the sliding action.

In another example, a user may slide a finger along the touch screeninterface (115) from a first location to a second location andthereafter stop by maintaining contact with the second location for athreshold period of time (e.g., one second). The detected continuedcontact with the second location may be used to determine that the userhas completed a swipe gesture.

In an embodiment, a sliding action (e.g., a swipe or a flick) may bedetected before the sliding action is completed. For example, aright-direction sliding gesture may be detected by detecting contact ata first location followed by contact at a second location that is to theright of the first location (or within a particular degree in the rightdirection). The user may continue the sliding gesture to a thirdlocation that is right of the second location, however, the direction ofthe sliding gesture may already be detected using the first location andthe second location.

In an embodiment, a flick gesture and a slide gesture (e.g., in the samedirection) may be mapped to the same video playback command.Accordingly, a device may be configured to detect either of the slidegesture or the flick gesture and identify the same video playbackcommand in response to the detected flick gesture or slide gesture.

In an embodiment, a flick gesture and a slide gesture (possibly in thesame direction) may be mapped to different commands. For example, aflick gesture to the left may correspond to a twenty second rewindcommand and a swipe gesture to the left may correspond to a command forselecting the previous bookmarked scene in a video. A scene may bebookmarked, for example, by a user or hard coded into a media recordingsuch as selectable scenes from a movie recorded on a Digital Video Disc(DVD).

In an embodiment, a slide gesture may be performed with multiple inputinstruments being used concurrently. For example, a user may slide twofingers across a touch screen interface at the same time. Further theuser may concurrently slide the two fingers in parallel (e.g., slidingtwo fingers in the same direction from left to right).

The term concurrently has referred to herein includes approximatelyconcurrent. For example, two fingers concurrently performing a parallelgesture may refer to two fingers of different lengths performing thesame gesture at slightly different times. For example, one finger maylag in time behind another finger for starting and/or finishing thegesture. Accordingly, the two fingers may start and finish the gestureat different start and/or finish times.

The term parallel as referred to herein include paths that are inapproximately the same direction. Two fingers performing a parallelmotion, as referred to herein, include a user dragging two fingersacross a touch screen interface in the same direction. Due to adifference in the length of the fingers or due to an angle of the hand,two or more fingers performing a parallel motion in the same generaldirection may differ in direction by a few degrees. In an embodiment,the paths along which two parallel gestures are performed may overlap.The term parallel, as referred to herein, may refer to any set of two ormore gestures that are performed in the same general direction.

4.0 GESTURE AREA(S)

In an embodiment, the touch screen interface (115) includes a gesturearea. A gesture area is at least a portion of the touch screen interface(115) that is configured to detect a gesture performed a user. Thegesture area may include the entire touch screen interface (115) or aportion of the touch screen interface (115). The gesture area maydisplay a blank box or one or more items. For example, the gesture areamay display a video. In another example, the gesture area may displayinformation on how to perform gestures.

In an embodiment, a gesture may be detected within a gesture areawithout a user's interaction with any visual objects that may bedisplayed in the gesture area. For example, a swipe gesture across acellular phone's touch screen interface (115) may be detected in agesture area that is an empty box on the touch screen interface. Inanother example, a progress indicator displayed in the gesture area isnot touched by a detected swipe gesture associated with a rewindcommand.

In an embodiment, any visual objects displayed within the gesture areaare not necessary for detecting a gesture or determining a commandrelated to the gesture. In an embodiment, any visual objects displayedwithin the gesture area are not selected or dragged by a fingerperforming the gesture.

In an embodiment, the touch screen interface (115) may include multiplegesture areas. A gesture detected within one gesture area may be mappedto a different command than the same gestured performed in a differentgesture area. A device may be configured to identify an area in which agesture is performed and determine an action based on the gesture andthe gesture area in which the action was performed.

In an embodiment, the gesture area of multiple gesture areas may beselected by a device when a gesture is detected across multiple gestureareas. The gesture area in which the gesture area was initiated may beidentified as the selected gesture area. For example, a user may begin aswipe gesture in a first gesture area and end the swipe gesture in asecond gesture area. In response to detecting that the swipe gesture wasinitiated in the first gesture area, the command mapped to the gestureand the first gesture area may be selected. In another example, agesture area in which the end of a sliding action is detected may beidentified as the intended gesture area. The selected or intendedgesture area may be then used to identify a command.

5.0 COMMANDS

In an embodiment, a gesture may be mapped to (or associated with) acommand. For example, a command mapped to a gesture may be a videoplayback command related to the playback of a video. The command may berelated to playback of a video on the device on which the command wasreceived or on a different device.

In an embodiment, a command may specify a video playing speed anddirection. For example, the command may select rewinding at a particularrewinding speed or fast-forwarding a particular fast-forwarding speed.Examples of other video playback commands include, but are not limitedto, pausing the playing of the video, resuming the playing of the video,replaying a played portion of the video, stopping playing of the video,stopping playing of the video and resuming playing of the video at aparticular playing position, playing the video in slow motion,frame-stepping through a video, playing the video from the beginning,playing one or more videos from a next playlist, playing the video froma particular scene forward, bookmarking a playing position in the video,stopping playing and resuming playing at a bookmarked position, orrating the video.

In an embodiment, a command may select a particular option out of a listof options. For example, a list of available media content may bedisplayed on a screen and the command may select particular mediacontent of the available media content. In another example, a list ofconfiguration settings may be displayed and the command may select aparticular setting for modification.

6.0 DETECTING A GESTURE WITHIN A GESTURE AREA

FIG. 2 illustrates a flow diagram for detecting a gesture within agesture area. One or more of the steps described below may be omitted,repeated, and/or performed in a different order. Accordingly, thespecific arrangement of steps shown in FIG. 2 should not be construed aslimiting the scope of the invention.

In one or more embodiments, detecting a gesture may include detectinginterface contact at an initial location that is a part of the detectedgesture (Step 202). The initial contact on the touch screen interfacemay be made with a user finger, a stylus, or any other item which may beused to perform a gesture on a touch screen interface. The initialcontact with the touch screen interface may involve a quick touch at theinitial location (e.g., a tap gesture) or a touch that is maintained atthe initial location for any period of time (e.g., a millisecond, asecond, two seconds, etc.). The initial contact with the touch screeninterface may be brief as may be made by a finger already moving in adirection. For example, a finger moving in the air without makingcontact, and thereafter during the moving making the initial contactwith a portion of the touch screen interface.

In an embodiment, the initial contact as referred to herein may includea finger (or other item) being close enough to a touch screen interfacethat the touch screen interface detects the finger. For example, whenusing a device including a capacitive system with a layer that storeselectrical charge, a part of the electrical charge may be transferred toa user where the user touches the touch screen interface or where a usersimply hovers close to the touch screen interface without touching.Accordingly, initial contact or maintained contact as referred to hereinmay include a user hovering a finger or other item over a touch screeninterface.

In an embodiment, the initial contact on the touch screen interface doesnot select any visual object displayed on touch screen interface. Theinitial contact may be made when no visual object is displayed. Theinitial contact may be made on top of a display of a visual objectwithout selecting the visual object. For example, the initial contactmay be made on a touch screen interface that is displaying auser-selected background image for the cellular phone. In anotherexample, the initial contact may be made on a blank screen. The initialcontact may be detected on a top of a television show being played on atablet.

In one or more embodiments, detecting a gesture may further includedetecting interface contact at additional locations, on the touch screeninterface (Step 204). For example, detecting a flick gesture or a swipegesture may include detecting interface contact at additional locationsin a chronological sequence along a path from the initial contactlocation. For example, interface contact may be detected continuously ina left-direction path away from an initial contact location on the touchscreen interface.

The contact along a path away from the location of the initial contactpoint may be referred to herein as a sliding gesture. In one or moreembodiments, a speed of the sliding gesture or a direction of thesliding gesture may be determined. For example, contact at two or morelocations on the interface, such as the initial contact point and asecond point along the path of the sliding gesture, may be used todetermine a direction and/or a speed of the sliding gesture. Contact atmultiple points may be used to calculate an acceleration of a slidinggesture.

In one or more embodiments, a gesture may be identified based on contactdetected at one or more locations on the touch screen interface (Step206). For example, detecting concurrent contact at three locations on aremote control interface followed by a release of contact at all threelocations may be identified as a three finger tap gesture. In anembodiment, detecting a gesture may include identifying a path alongwhich contact was detected on the touch screen interface. For example, acircle gesture may be identified in response to detecting contact alonga circular path on a touch screen interface. A flick gesture or a swipegesture may be identified based on contact points in a chronologicalsequence on a touch screen interface.

In one or more embodiments, identifying a gesture may includedetermining a number of concurrent parallel gestures (Step 208). Forexample, initial contact may be detected concurrently at multiplelocations on a touch screen interface. Subsequent to the initial contactat each initial location, contact along paths beginning from the initiallocations may be detected. If the paths are determined to be parallel,the number of paths may be identified to determine the number ofconcurrent parallel gestures.

In an embodiment, a number of concurrent parallel gestures may bedetermined based on the number of paths that match a knownconfiguration. For example, if a path has at least a first contact pointand a subsequent second contact point to the right within ten degreesfrom a horizontal line from the first contact point, the path may bedetermined to correspond to a sliding gesture to the right. The numberof detected gestures that correspond to paths that match the samecriteria within a particular time period may be counted to determine thenumber of concurrent parallel gestures. In an embodiment, other methodsnot described herein may be used for determining the number ofconcurrent parallel gestures.

In an embodiment, a command is determined based on an identified gesture(Step 210). The command may be determined while the gesture is stillbeing performed or after the gesture is completed.

In an embodiment, determining a command may include determining that aparticular detected gesture is mapped to a command in a database. Forexample, a two fingered swipe to the right may be queried in a commanddatabase to identify a command associated with the two fingered swipe.In another example, a two fingered flick toward the bottom of thegesture area may be associated with a command for selecting the secondmenu item out of items currently displayed in a menu.

In an embodiment, a number of parallel fingers in a command may be usedto determine a playback speed for the playing of multi-media content.For example, detection of two parallel gestures may be mapped to acommand for playback speed which is two times a normal playback speed.

In an embodiment, a direction of gesture command may be combined with anumber of parallel fingers in the gesture command to determine theplayback command. For example, two fingers swiped concurrently from theright side of the screen to the left side of the screen may be mapped torewind at two times a normal speed. In another example, two fingersswiped concurrently from the left side of the screen to the right sideof the screen may be mapped to fast-forward at a speed that is twice thenormal playback speed (without fast-forwarding).

In an embodiment, a command may include resuming playing of a video atparticular bookmarks (e.g., user defined bookmarks or manufacturerdefined bookmarks). A number of fingers used to perform a concurrentparallel gesture may be used to select the bookmark. For example, inresponse to detecting a two-fingered flick downward, the playing of avideo may be resumed at the second bookmark from a current playingposition.

In an embodiment, determining a command may include identifying thedevice corresponding to the command. For example, a device related tothe command may be identified based on the gesture and/or the gesturearea in which the gesture was detected.

In an embodiment, an action corresponding to the command is performed(Step 212). The action may be performed by a device that detects thecommand. For example, if a gesture for a fast-forward command isdetected on a hand-held touch screen phone that is playing a video, thehand-held touch screen phone play the video in fast-forward mode.

In an embodiment, an action corresponding to the command may includetransmitting information related to the command to another device. Forexample, a gesture may be detected on a touch screen remote control.Information related to the gesture (e.g., information identifying thegesture or information identifying a command associated with thegesture) may then be transmitted to a digital video disc player. Thedigital video disc player may then perform a corresponding action. Ifthe command was for pausing the playing of a video, the digital videodisc player may pause the playing of the video on a display screen.

7.0 EXAMPLE GESTURES AND COMMANDS

FIG. 3 illustrates an example screen shot for an input device configuredto detect gestures. The gestures, commands, mapping between gestures andcommands, gesture areas, visual objects, and any other items discussedin relation to FIG. 3 are examples and should not be construed aslimiting in scope. One or more of the items described in relation toFIG. 3 may not be necessarily implemented and other items described maybe implemented in accordance with one or more embodiments.

FIG. 3 illustrates an example interface (300) with a circular gesturearea (305) and a square gesture area (310). Any gestures detected incircular gesture area (305) are mapped to navigation commands. Forexample, a two fingered tap detected in circular gesture area (305) maybe associated with a command selecting a second item on any currentlydisplayed menu. If the second item is a folder, the items within thefolder may be displayed in response to detecting the two fingered tap.

In an embodiment, square gesture area (310) may identify commands thatare associated with one or more gestures detected within the squaregesture area (310). For example, the square gesture area (310) mayinclude graphics illustrating that a single finger swipe gesture to theleft corresponds to a rewind command, a single finger tap gesturecorresponds to a pause command, a single finger swipe gesture to theright corresponds to a fast-forward command, a two fingered swipegesture to the left corresponds to a ten second rewind, a two fingeredtap gesture corresponds to show motion playback command, and a twofingered swipe to the right corresponds to skip to next bookmarkcommand.

In an embodiment, the example interface (300) may include a progressindicator (315) which is separate from the circular gesture area (305)and the square gesture area (310). The progress indicator (315) mayinclude a current playing position of the video, bookmarks, a currentplayback speed, etc. For example, the progress indicator (315) mayinclude a symbol representing a current playback speed (e.g., play, fastforward at lx, pause, rewind at 2×, etc.).

In an embodiment, the symbol may be displayed in response to a command.For example, in response to a rewind at 3× command, a symbol indicating3× rewind may displayed while rewinding multimedia content at 3× isperformed by displaying frames in reverse at three times the normalplayback speed. However, the progress indicator (315) may notnecessarily be selected by any gesture associated with a video playbackcommand. In an embodiment, no visual objects within example interface(300) are necessarily selected when a user is performing a gesturewithin the example interface (300).

In an embodiment, the example interface (300) may also include a tool(e.g., a drop down box) to select a particular media device to becontrolled by detected gestures. In an embodiment, the example interface(300) may include an option to switch between input mechanisms (e.g.,gesture based input, buttons, text box, radio boxes, etc.).

8.0 REMOTE CONTROL USE EXAMPLE

In an embodiment, a remote control device communicates with a mediadevice (e.g., a digital video recorder, a digital video disc player, amedia management device, a video recorder, a blu-ray player, etc.). Theremote control device may communicate with the media device over wiredand/or wireless communication segments. For example, the remote controldevice may communicate over a network (e.g, internet, intranet, etc.),via radio communication, over Bluetooth, via infrared, etc.

In an embodiment, a remote control displays a progress indicator (315)as shown in the screen shot (300) of FIG. 3. The progress indicator(315) may indicate a playing position of multimedia content beingdisplayed on a separate multimedia device. The progress indicator (315)may display an exact playing position or an approximate playingposition. For example, the progress indicator (315) may include a slider(320) displayed along a trickplay bar (330) to indicate the playingposition. In an embodiment, a particular playing position may beindicated by a time (e.g., 8:09). The time may indicate, for example,the actual streaming time of the currently played content or mayindicate an offset from the starting point of the content.

In an embodiment, information related to the playing position of themultimedia content may be obtained from a media device (e.g., a digitalvideo recorder, a cable box, a computer, a media management device, adigital video disc player, multimedia player, audio player, etc.). Forexample, a remote control device communicatively coupled with a mediadevice may be configured to receive frame information related to theparticular frame being displayed (played) by the media device. In anembodiment, the media device may periodically send the remote controldevice the frame information. Alternatively, the remote control devicemay periodically request the frame information from the media device.The remote device uses the information to position the slider (320)along the trickplay bar (330). The remote control device can alsoreceive information from the media device indicating the extent of thecache bar (325) which indicates the amount of multimedia content storedor recorded by the media device. If the media device is in the processof recording or caching a multimedia content, the cache bar (325) willincrease in size as the media device records or caches more content. Ifthe media device is playing a recorded multimedia content, then thecache bar (325) extends the length of the trickplay bar (330).

Another example may involve the remote control device being configuredto receive a time stamp closest to the frame being displayed. The remotecontrol device may also be configured to use a step function, e.g., nextframe or previous frame from the time stamp if no frame is an exactmatch to the time stamp. Another example may include the remote controldevice continuously receiving images (e.g., bitmap, displayinstructions, etc.) from the media device of the progress indicator todisplay on the remote control device. In an embodiment, the remotecontrol device may include a particular starting position and a displayrate for use by the remote control device to determine the playingposition of the multimedia content. For example, a digital videorecorder may transmit an initial playing position in the playing of themultimedia content to the remote control device with a rate of progress(e.g., change of the slider (320) per unit of time, frame rate, etc.).The remote control device may use the information to first display aprogress indicator based on the initial playing position and may thencompute the subsequent positions as a function of time.

In an embodiment, the slider (320) becomes out of sync with a displayedvideo when a trickplay function is performed (e.g., when a ten secondrewind is performed). In response to a trickplay function, updatedinformation regarding a new playing position may be provided to theremote control device.

In an embodiment, the remote control device may further receive updatesselecting specific playing positions or indicating changes in the rateof progress. For example, a user may submit one or more commands topause the playing of multimedia content at a current playing position,then skip back 10 seconds before the current playing position and thenresume playing. In this case, a media device may provide information tothe remote control device to pause the slider (320), display a newplaying position corresponding to 10 seconds before the current playingposition by moving the slider (320), and then resume periodicallyupdating the slider (320).

In an embodiment, the slider (320) may be updated when the remotecontrol device is activated. For example, when a user picks up theremote control device or touches the remote control device, the remotecontrol device may request playing position information from a mediadevice. For example, the remote control device may include anaccelerometer configured to detect motion and/or a touch screeninterface configured to detect touch. In response, the media device mayprovide playing position information to the remote control device. Theremote control device may then display the slider (320) indicating acurrent playing position of multimedia content based on the playing onthe position information received from the media device.

In an embodiment, information related to the playing position of themultimedia content may be continuously received by the remote controldevice for the remote control device to constantly update the slider(320). In another embodiment, the information related to the playingposition of the multimedia content may be periodically received and theremote control device may update the slider each time the information isreceived.

In an embodiment, the remote control device may transmit the multimediacontent to the multimedia device for display by the multimedia device.For example, the remote control device may obtain a video stream overthe internet and send the video stream to a multimedia device fordisplay on the multimedia device. In this example, the remote controldevice may determine the display position of the slider (320) based onplaying position information determined by the remote control deviceitself. For example, the remote control device may compute the playingposition information based on a frame being sent to the multimediadevice from the remote control device.

9.0 EXAMPLE EMBODIMENTS

In an embodiment, a method comprises detecting a slide gesture, in aparticular area on a touch screen interface of a device, from a firstlocation in the particular area to a second location in the particulararea; identifying a video playback command based at least on the slidegesture; performing an action associated with the video playbackcommand; wherein the method is performed by at least one device.

In an embodiment, the sliding gesture is detected without detectingselection of any video progress indicator displayed within theparticular area. The slide gesture may be detected in the particulararea while displaying at least a portion of the video in the particulararea. The slide gesture may be detected in the particular area whiledisplaying information on how to perform one or more gestures in theparticular area.

In an embodiment, identifying the video playback command is furtherbased on the particular area, in which the slide gesture was detected,from a plurality of areas on the touch screen interface.

In an embodiment, performing the action comprises a first device sendinginformation to a second device, the information based on the videoplayback command. Performing the action associated with the video maycomprise performing the action on a same device as the device detectingthe slide gesture. The video playback command may select a playing speedand direction.

In an embodiment, the slide gesture comprises a swipe gesture from thefirst location to a second location. The slide gesture may comprise aflick gesture starting at the first location.

In an embodiment, the video playback command is for one or more of:pausing the playing of the video; resuming the playing of the video;replaying a played portion of the video; stopping playing of the video;stopping playing of the video and resuming playing of the video at aparticular playing position; playing the video in slow motion; playingthe video from the beginning; playing one or more videos from a nextplaylist; playing the video from a particular scene forward; bookmarkinga playing position in the video; stopping playing and resuming playingat a bookmarked position; or rating the video.

In an embodiment, a method comprises concurrently detecting a pluralityof parallel gestures on a touch screen interface of a device;determining a number of the plurality of parallel gestures; selecting acommand from a plurality of commands based on the number of theplurality of parallel gestures; performing an action associated with thecommand.

In an embodiment, selecting the command comprises selecting a menuoption based on the number of the plurality of parallel gestures. Theplurality of parallel gestures may comprise a plurality of parallelsliding gestures performed in a same direction.

In an embodiment, determining the number of the plurality of parallelgestures comprises determining a number of tap gestures concurrentlyperformed on the touch screen interface.

Although specific components are recited herein as performing the methodsteps, in other embodiments agents or mechanisms acting on behalf of thespecified components may perform the method steps. Further, althoughsome aspects of the invention are discussed with respect to componentson a system, the invention may be implemented with componentsdistributed over multiple systems. Embodiments of the invention alsoinclude any system that includes the means for performing the methodsteps described herein. Embodiments of the invention also include acomputer readable medium with instructions, which when executed, causethe method steps described herein to be performed.

10.0 HARDWARE OVERVIEW

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 4 is a block diagram that illustrates a computersystem 400 upon which an embodiment of the invention may be implemented.Computer system 400 includes a bus 402 or other communication mechanismfor communicating information, and a hardware processor 404 coupled withbus 402 for processing information. Hardware processor 404 may be, forexample, a general purpose microprocessor.

Computer system 400 also includes a main memory 406, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 402for storing information and instructions to be executed by processor404. Main memory 406 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 404. Such instructions, when stored innon-transitory storage media accessible to processor 404, rendercomputer system 400 into a special-purpose machine that is customized toperform the operations specified in the instructions.

Computer system 400 further includes a read only memory (ROM) 408 orother static storage device coupled to bus 402 for storing staticinformation and instructions for processor 404. A storage device 410,such as a magnetic disk or optical disk, is provided and coupled to bus402 for storing information and instructions.

Computer system 400 may be coupled via bus 402 to a display 412, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 414, including alphanumeric and other keys, is coupledto bus 402 for communicating information and command selections toprocessor 404. Another type of user input device is cursor control 416,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 404 and forcontrolling cursor movement on display 412. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

Computer system 400 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 400 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 400 in response to processor 404 executing one or more sequencesof one or more instructions contained in main memory 406. Suchinstructions may be read into main memory 406 from another storagemedium, such as storage device 410. Execution of the sequences ofinstructions contained in main memory 406 causes processor 404 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 410.Volatile media includes dynamic memory, such as main memory 406. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 402. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 404 for execution. For example,the instructions may initially be carried on a magnetic disk or solidstate drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 400 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 402. Bus 402 carries the data tomain memory 406, from which processor 404 retrieves and executes theinstructions. The instructions received by main memory 406 mayoptionally be stored on storage device 410 either before or afterexecution by processor 404.

Computer system 400 also includes a communication interface 418 coupledto bus 402. Communication interface 418 provides a two-way datacommunication coupling to a network link 420 that is connected to alocal network 422. For example, communication interface 418 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 418 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, communication interface 418sends and receives electrical, electromagnetic or optical signals thatcarry digital data streams representing various types of information.

Network link 420 typically provides data communication through one ormore networks to other data devices. For example, network link 420 mayprovide a connection through local network 422 to a host computer 424 orto data equipment operated by an Internet Service Provider (ISP) 426.ISP 426 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 428. Local network 422 and Internet 428 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 420and through communication interface 418, which carry the digital data toand from computer system 400, are example forms of transmission media.

Computer system 400 can send messages and receive data, includingprogram code, through the network(s), network link 420 and communicationinterface 418. In the Internet example, a server 430 might transmit arequested code for an application program through Internet 428, ISP 426,local network 422 and communication interface 418.

The received code may be executed by processor 404 as it is received,and/or stored in storage device 410, or other non-volatile storage forlater execution.

The received code may be executed by processor 604 as it is received,and/or stored in storage device 610, or other non-volatile storage forlater execution. In an embodiment, an apparatus is a combination of oneor more hardware and/or software components described herein. In anembodiment, a subsystem for performing a step is a combination of one ormore hardware and/or software components that may be configured toperform the step.

11.0 EXTENSIONS AND ALTERNATIVES

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. Thus, the sole and exclusive indicatorof what is the invention, and is intended by the applicants to be theinvention, is the set of claims that issue from this application, in thespecific form in which such claims issue, including any subsequentcorrection. Any definitions expressly set forth herein for termscontained in such claims shall govern the meaning of such terms as usedin the claims. Hence, no limitation, element, property, feature,advantage or attribute that is not expressly recited in a claim shouldlimit the scope of such claim in any way. The specification and drawingsare, accordingly, to be regarded in an illustrative rather than arestrictive sense.

1. A method, comprising: detecting a gesture that has been made on atouch screen interface of a first device; selecting a device, from aplurality of devices, and a command, from a plurality of commands, basedat least on one or more of: which gesture was identified, and/or atwhich area of the touch screen interface the gesture was detected;causing performance of the selected command at the selected device. 2.The method as recited in claim 1, wherein the detected gesture is one ofa plurality of concurrent parallel sliding gestures performed in a samedirection, wherein the command is further selected from the plurality ofcommands, or the device is further selected from the plurality ofdevices, based on the plurality of concurrent parallel sliding gestures.3. The method as recited in claim 1, wherein the detected gesture is oneof a number of tap gestures concurrently performed on the touch screeninterface, wherein the command is further selected from the plurality ofcommands, or the device is further selected from the plurality ofdevices, based on the number of tap gestures.
 4. The method of claim 1,further comprising: displaying, at the first device, a progressindicator indicating a current playback position within the multimediacontent played on the selected device; updating the progress indicatorto reflect a new playback position of the multimedia content at theselected device in response to performance of the selected command;wherein the selected command is a multimedia playback command formultimedia content playing at the selected device.
 5. The method ofclaim 4, wherein the first device updates the progress indicatorresponsive to periodic messages from the selected device.
 6. The methodof claim 4, wherein the first device updates the progress indicator byrequesting progress updates from the selected device, based in part oninput from an accelerometer or touch screen indicating that the firstdevice is in active use.
 7. The method of claim 1, further comprising:displaying a menu of options, each of the options having an optionnumber, each of the options corresponding to a different command of theplurality of commands; wherein selecting the command comprises selectingwhich of the plurality of commands corresponds to a particular option,of the options, that has an option number equivalent to how manyparallel gestures were being made concurrently when the gesture wasdetected.
 8. The method of claim 1, wherein the plurality of devicesinclude both the first device and a device that is remote from the firstdevice and connected to the first device via a wireless connection. 9.The method of claim 1, further comprising: identifying a plurality ofgesture areas at which gestures may be made on the touch screeninterface; selecting the device from the plurality of devices based ondetermining in which of the plurality of gesture areas the detectedgesture was made.
 10. The method of claim 9, further comprisingoutputting, in different gesture areas of the plurality of gestureareas, different graphical items, the graphical items including one ormore of: a video, a bitmap, or instructional information.
 11. Anon-transitory computer readable medium comprising a sequence ofinstructions, which when executed by one or more processors, causeperforming steps comprising: detecting a gesture that has been made on atouch screen interface of a first device; selecting a device, from aplurality of devices, and a command, from a plurality of commands, basedat least on one or more of: which gesture was identified, and/or atwhich area of the touch screen interface the gesture was detected;causing performance of the selected command at the selected device. 12.The non-transitory computer readable medium as recited in claim 11,wherein the detected gesture is one of a plurality of concurrentparallel sliding gestures performed in a same direction, wherein thecommand is further selected from the plurality of commands, or thedevice is further selected from the plurality of devices, based on theplurality of concurrent parallel sliding gestures.
 13. Thenon-transitory computer readable medium as recited in claim 11, whereinthe detected gesture is one of a number of tap gestures concurrentlyperformed on the touch screen interface, wherein the command is furtherselected from the plurality of commands, or the device is furtherselected from the plurality of devices, based on the number of tapgestures.
 14. The non-transitory computer readable medium of claim 11,wherein the steps further comprise: displaying, at the first device, aprogress indicator indicating a current playback position within themultimedia content played on the selected device; updating the progressindicator to reflect a new playback position of the multimedia contentat the selected device in response to performance of the selectedcommand; wherein the selected command is a multimedia playback commandfor multimedia content playing at the selected device.
 15. Thenon-transitory computer readable medium of claim 14, wherein the firstdevice updates the progress indicator responsive to periodic messagesfrom the selected device.
 16. The non-transitory computer readablemedium of claim 14, wherein the first device updates the progressindicator by requesting progress updates from the selected device, basedin part on input from an accelerometer or touch screen indicating thatthe first device is in active use.
 17. The non-transitory computerreadable medium of claim 11, wherein the steps further comprise:displaying a menu of options, each of the options having an optionnumber, each of the options corresponding to a different command of theplurality of commands; wherein selecting the command comprises selectingwhich of the plurality of commands corresponds to a particular option,of the options, that has an option number equivalent to how manyparallel gestures were being made concurrently when the gesture wasdetected.
 18. The non-transitory computer readable medium of claim 11,wherein the plurality of devices include both the first device and adevice that is remote from the first device and connected to the firstdevice via a wireless connection.
 19. The non-transitory computerreadable medium of claim 11, wherein the steps further comprise:identifying a plurality of gesture areas at which gestures may be madeon the touch screen interface; selecting the device from the pluralityof devices based on determining in which of the plurality of gestureareas the detected gesture was made.
 20. The non-transitory computerreadable medium of claim 19, wherein the steps further compriseoutputting, in different gesture areas of the plurality of gestureareas, different graphical items, the graphical items including one ormore of: a video, a bitmap, or instructional information.
 21. A devicecomprising: one or more processors; logic, implemented at least partlyby hardware, configured to detect a gesture that has been made on atouch screen interface of a first device; logic, implemented at leastpartly by hardware, configured to select a device, from a plurality ofdevices, and a command, from a plurality of commands, based at least onone or more of: which gesture was identified, and/or at which area ofthe touch screen interface the gesture was detected; logic, implementedat least partly by hardware, that causes performance of the selectedcommand at the selected device; wherein the selected command is amultimedia playback command for multimedia content playing at theselected device.
 22. The device as recited in claim 21, wherein thedetected gesture is one of a plurality of concurrent parallel slidinggestures performed in a same direction, wherein the command is furtherselected from the plurality of commands, or the device is furtherselected from the plurality of devices, based on the plurality ofconcurrent parallel sliding gestures.
 23. The device as recited in claim21, wherein the detected gesture is one of a number of tap gesturesconcurrently performed on the touch screen interface, wherein thecommand is further selected from the plurality of commands, or thedevice is further selected from the plurality of devices, based on thenumber of tap gestures.
 24. The device of claim 21, further comprising:logic, implemented at least partly by hardware, configured to display,at the first device, a progress indicator indicating a current playbackposition within the multimedia content played on the selected device;logic, implemented at least partly by hardware, configured to update theprogress indicator to reflect a new playback position of the multimediacontent at the selected device in response to performance of theselected command.
 25. The device of claim 24, wherein the first deviceupdates the progress indicator responsive to periodic messages from theselected device.
 26. The device of claim 24, wherein the first deviceupdates the progress indicator by requesting progress updates from theselected device, based in part on input from an accelerometer or touchscreen indicating that the first device is in active use.
 27. The deviceof claim 21, further comprising: logic, implemented at least partiallyby hardware, configured to display a menu of options, each of theoptions having an option number, each of the options corresponding to adifferent command of the plurality of commands; wherein selecting thecommand comprises selecting which of the plurality of commandscorresponds to a particular option, of the options, that has an optionnumber equivalent to how many parallel gestures were being madeconcurrently when the gesture was detected.
 28. The device of claim 21,wherein the plurality of devices include both the first device and adevice that is remote from the first device and connected to the firstdevice via a wireless connection.
 29. The device of claim 21, furthercomprising: logic, implemented at least partially by hardware,configured to identify a plurality of gesture areas at which gesturesmay be made on the touch screen interface; logic, implemented at leastpartially by hardware, configured to select the device from theplurality of devices based on determining in which of the plurality ofgesture areas the detected gesture was made.
 30. The device of claim 29,further comprising logic, implemented at least partially by hardware,configured to output, in different gesture areas of the plurality ofgesture areas, different graphical items, the graphical items includingone or more of: a video, a bitmap, or instructional information.